Flat panel, color displays for displaying information, including images, text, and graphics are widely used. These displays may employ any number of known technologies, including liquid crystal light modulators, plasma emission, electro-luminescence (including organic light-emitting diodes), and field emission. Such displays include entertainment devices such as televisions, monitors for interacting with computers, and displays employed in hand-held electronic devices such as cell phones, game consoles, and personal digital assistants. In these displays, the resolution of the display is always a critical element in the performance and usefulness of the display. The resolution of the display specifies the quantity of information that can be usefully shown on the display and the quantity of information directly impacts the usefulness of the electronic devices that employ the display.
However, the term “resolution” is often used or misused to represent any number of quantities. Common misuses of the term include referring to the number of light-emitting elements or to the number of full-color groupings of light-emitting elements (typically referred to as pixels) as the “resolution” of the display. This number of light-emitting elements is more appropriately referred to as the addessability of the display. Within this document, we will use the term “addressability” to refer to the number of light-emitting elements per unit area of the display device. A more appropriate definition of resolution is to define the size of the smallest element that can be displayed with fidelity on the display. One method of measuring this quantity is to display the narrowest possible, neutral (e.g., white) horizontal or vertical line on a display and to measure the width of this line or to display an alternating array of neutral and black lines on a display and to measure the period of this alternating pattern. Note that using these definitions, as the number of light-emitting elements increases within a given display area, the addressability of the display will increase while the resolution, using this definition, generally decreases. Therefore, counter to the common use of the term “resolution”, the quality of the display is generally improved as the resolution becomes finer in pitch or smaller.
The term “apparent resolution” refers to the perceived resolution of the display as viewed by the user. Although, methods for measuring the physical resolution of the display device are typically designed to correlate with apparent resolution, it is important to note that this does not always occur. At least two important conditions under which the physical measurement of the display device does not correlate with apparent resolution exist. The first of these occur when the physical resolution of the display device is small enough that the human visual system is unable to resolve changes in physical resolution (i.e., the apparent resolution of the display becomes eye-limited). The second condition occurs when the measurement of the physical resolution of the display is performed for only the luminance channel but not performed for resolution of the color information while the display actually has a different resolution within each color channel.
Addressability in most flat-panel displays, especially active-matrix displays, is limited by the need to provide signal busses and electronic control elements in the display. Further in many flat panel displays, including Liquid Crystal Displays (LCDs) and bottom-emitting Electro-Luminescent (EL) displays, the electronic control elements are required to share the area that is required for light emission or transmission. In these technologies, the more such busses and control elements that are needed, the less area in the display is available for actual light-emitting areas. Depending upon the technology, reduction of the area of the light-emitting area can reduce the efficiency of light output, as is the case for LCDs, or reduce the brightness and/or lifetime of the display device, as is the case for EL displays. Regardless of whether the area required for patterning busses and control elements competes with the light-emitting area of the display, the decrease in buss and control element size that occur with increases in addressability for a given display generally require more accurate, and therefore more complex, manufacturing processes and can result in greater number of defective panels, decreasing yield rate and increasing the cost of marketable displays. Therefore, from a cost and manufacturing complexity point of view, it is generally advantageous to be able to provide a display with lower addressability. This desire is, of course, in conflict with the need to provide higher apparent resolution. Therefore, it would be desirable to provide a display that has relatively low addessability but that also provides high apparent resolution.
It has been known for many years that the human eye is more sensitive to luminance in a scene than to color. In fact, current understanding of the visual system includes the fact that processing is performed within or near the retina of the human eye that converts the signal that is generated by the photoreceptors into a luminance signal, a red/green difference signal and a blue/yellow difference signal. Each of these three signals have different resolution as depicted by the modulation threshold curves shown in FIG. 1 for a given user population and illumination level. As shown, the luminance channel can resolve the finest detail as indicated by the fact that the modulation threshold curve for the luminance signal 2 has the highest spatial frequency cutoff, the modulation threshold for the red/green signal 4 has the second highest spatial frequency cutoff and the blue/yellow signal 6 has the lowest spatial frequency cutoff and that the cutoff for the blue/yellow signal is on the order of one fourth the cutoff for the luminance signal. It is further notable that while the human visual system is sensitive to relatively high frequency spatial information in the luminance channel, it is less sensitive to very low spatial frequency information in the luminance channel. And while the human visual system is not as sensitive to high spatial frequency in the chrominance channels as in the luminance channels, it can be quite sensitive to even very low spatial frequency in the chrominance channels.
This difference in sensitivity is well appreciated within the imaging industry and has been employed to provide lower cost systems with high perceived quality within many domains, most notably digital camera sensors and image compression and transmission algorithms. For example, since green light provides the preponderance of luminance information in typical viewing environments, digital cameras typically employ two green sensitive elements for every red and blue sensitive element and interpolate intermediate luminance values for the missing colored elements within each color plane. In typical image compression and transmission algorithms, image signals are converted to a luminance/chrominance representation and the chrominance channels undergo significantly more compression than the luminance channel.
Similarly, this fact has been used in display devices to provide high apparent resolution for a reduced addressability. Takashi et al. in U.S. Pat. No. 5,113,274, entitled “Matrix-type color liquid crystal display device”, has proposed the use of displays having two green for every red and blue light-emitting element. While such an array of light-emitting elements can perform well for displays with a very high addressability, it is important that the red light-emitting elements typically provide approximately 30 percent of the luminance. Therefore, under certain conditions, such as when displaying flat fields of red, it is possible to see artifacts (e.g., a red and black checkerboard pattern in areas that are intended to be perceived as a flat field red) that occur because of the scarcity of the red light-emitting elements within the array. Therefore, it is important to understand that in displays it is not only the size or the frequency of light-emitting elements that are important in order to understand the quality of the display device but also the space between the light-emitting elements. Therefore, the relative location of the different light-emitting elements within the array can produce displays with significantly different appearance. For example, when using arrays such as proposed by Takashi, it is very important that the position of the red and blue light-emitting elements be alternated within each pair of rows and columns of the display device as this significantly reduces the appearance of artifacts such as the checkerboard pattern. It is also appreciated in the art that by offsetting the high luminance elements within an array of light-emitting elements, the perceived artifacts may be adjusted. For example it is known to offset alternate rows of red, green, and blue light-emitting elements on low resolution pictorial displays (a pixel pattern commonly referred to as the delta pattern since pixels are formed from red, green, and blue elements that are arranged in triangles) to create a higher perceived quality display since by offsetting the high luminance green elements on successive rows, the images that are presented have a “smoother” appearance. It is also recognized, however, that these effects can be quite image content dependent and therefore, displays that are designed to present text do not offset the position of light-emitting elements within alternate rows as this pixel arrangement creates the appearance of ragged edges on high contrast vertical lines, which occur frequently in text and this ragged appearance (commonly referred to as “jaggies”) can be quite disturbing to the user.
In addition to higher perceived quality, the introduction of more high luminance light-emitting elements into a display can have other positive effects. For example, within the field of Organic Light Emitting Diodes (OLEDs), it is known to introduce more than three light-emitting elements where the additional light-emitting elements have higher luminance efficiency, resulting in a display having higher luminance efficiency. Such displays have been discussed by Miller et al. in US Patent Application Publication 2004/0113875 entitled “Color OLED display with improved power efficiency” and US Patent Application Publication 2005/0212728 also entitled “Color OLED display with improved power efficiency”.
This fact has been used in a variety of ways to optimize the frequency response of imaging systems. For example, relative sensitivities of the human eye to different color channels have recently been used in the liquid crystal display (LCD) art to produce displays having subpixels with broad band emission to increase perceived resolution. For example, US Patent Application 2005/0225574 and US Patent Application 2005/0225575, each entitled “Novel subpixel layouts and arrangements for high brightness displays” provide various subpixel arrangements such as the one shown in FIG. 2. FIG. 2 shows a portion of a prior art display 10 as discussed within these disclosures. Of importance in this subpixel arrangement is the existence of a high-luminance subpixel, such as the white subpixel 12 that allows more of the white light generated by the LCD backlight to be transmitted to the user than the traditional filtered RGB subpixels (14, 16, and 18) and the fact that each row in the subpixel arrangement contains all colors of subpixels, makes it possible to produce a line of any color using only one row of subpixels. Similarly, every pair of columns within the subpixel arrangement contain all colors of subpixels within the display, making it possible to produce a line of any color using only two columns of subpixels. Therefore, when the LCD is driven correctly, it can be argued that the vertical resolution of the device is equal to the height of one row of subpixels and the horizontal resolution of the device is equal to the width of two columns of subpixels, even though it realistically requires more subpixels than the two subpixels at the intersection of such horizontal and vertical lines to produce a full-color image. However, since each pair of subpixels at the junction of such horizontal and vertical lines contain at least one high luminance subpixel (typically green 16 or white 12), each pair of light-emitting elements provide a relatively accurate luminance signal within each pair of subpixels, providing a high-resolution luminance signal. It is important to note that in arrangements of light-emitting elements such as these, as well as those discussed by Takashi, there are more high-luminance light-emitting elements than there are repeating patterns of light-emitting elements that are capable of producing a full-color image. Therefore, by using arrangements of light-emitting elements such as these, it is possible to display a luminance pattern with a higher spatial frequency than would be possible if each luminance signal was to be rendered to each repeating pattern of light-emitting elements. However, to achieve this goal, a proper rendering algorithm must be provided to provide this higher resolution rendering without creating significant color artifacts.
Many input image signals may be used to encode and transmit a full-color image for display. For example, an input image may be described in common RGB color spaces such as sRGB or in luminance/chrominance spaces such as YUV, L*a*b*, or YIQ. In any case, the input display signal must be converted to a signal suitable for driving the native display light-emitting elements. This conversion may involve steps such as conversion of a three-color input image signal to a signal to drive an array of four or more colors of light-emitting elements as described in U.S. Pat. No. 6,897,876 issued May 24, 2005. This conversion may also comprise methods such as subpixel interpolation like those described in US Patent Application 2005/0225563, entitled “Subpixel rendering filters for high brightness subpixel layouts”, which allows an input image signal that is intended for display on an arrangement of subpixels to be interpolated such that the input data is more appropriately matched to an alternate arrangement of subpixels. While subpixel interpolation methods known in the art allow different spatial filtering operations to be performed on signals that are intended for display on subpixels having different colors, they do not fully allow the optimization of the signal to take advantage of the difference in the human visual system's sensitivities to luminance and chrominance information. Specifically, the known subpixel interpolation techniques generally apply a static, typically even, function to the image information where this function is an averaging function that smoothes the image content. As such, the known subpixel interpolation algorithms generally blur the image content. To counter the blur introduced by such a subpixel interpolation algorithm, luminance bearing color channels must then be sharpened to boost the low frequency content in order to compensate for the lost high frequency content that occurs as a result of subpixel interpolation as discussed within this application, increasing the number of image processing steps that must be conducted or increasing the necessary size of the convolution kernel which then requires more image information to be buffered and increases the computational complexity of the process.
Pixel fault masking algorithms have also been proposed in RGBW systems as described in WO 03/100756, entitled “Pixel Fault Masking” which render information to neighboring light-emitting elements when one element is incapable of producing light due to manufacturing defects. As described in this application, these algorithms are known to consider information to be displayed by light-emitting elements that are neighbors to a faulty light-emitting element to form a weighting function in an optimization algorithm that attempts to minimize perceived error. As such these algorithms may render information to light-emitting elements that surround a faulty light-emitting element by applying a function that is dependent upon the content of the image to be displayed. However, since the formation of this rendering function requires an optimization problem to be solved, which can be quite compute intensive. Further, as it is a feature that the “problem only needs to be solved for the defect pixels” as taught therein, of which there are typically only tens of defect pixels in a display having millions of subpixels, there is no teaching of any process applicable to the rendering of a full-color image to each light-emitting element within a display.
It is known in the art to perform separate manipulations on luminance than on chrominance-encoded signals. For example, U.S. Pat. No. 5,987,169, entitled “Method for improving text resolution in images with reduced chromatic bandwidth” recognizes that some compression means provide excessive blurring to high spatial frequency, high luminance chrominance information, resulting in text or other high spatial frequency image objects that appear blurred. To overcome this problem, this patent discusses reducing the chrominance signal for highly chromatic text displayed on bright (white) backgrounds.
US Patent Application 2002/0154152, entitled “Display apparatus, display method and display apparatus controller” describes a display having red, green, and blue elements or subpixels which form full-color pixels. This display receives an input image signal, converts the signal to a luminance and chrominance signal, then renders the luminance information to the subpixel level but renders the chroma information to the pixel level, thus the luminance signal is represented at a higher spatial frequency than the chrominance signal, thereby providing a higher perceived resolution without significant lower frequency chromatic artifacts. To obtain optimal performance according to this invention, it is necessary that the input image signal address a number of spatial locations equal to the number of subpixels in the display device. However, this patent application is deficient in that because the arrangements of light-emitting elements that are discussed include only one high luminance light-emitting element per pixel, the subpixel arrangement limits the usefulness of this approach since the low luminance red and blue subpixels discussed in this patent application actually present little luminance information and therefore are incapable of rendering a significant portion of the higher addressability luminance information that is present in the input signal. Further, this patent only employs linear transforms to convert from one three-channel image representation to a second three-channel representation and as such can not be applied when converting an input three-color signal to a four-or-more output color signal. Further, the disclosure assumes that a perfect rendering can be obtained without luminance or chrominance error, while in practice some degree of luminance and/or chrominance error will often practically be present and an appropriate tradeoff must be made between these errors. Finally, the method ignores the fact that different tradeoffs between localized luminance and chrominance error may be made depending upon the spatial content of the image.
U.S. Pat. No. 5,793,885 entitled “Computationally efficient low artifact system for spatially filtering digital color images” also discusses converting an input image to a luminance and chrominance domain and then applying sharpening to only the luminance channel in the input RGB image. By applying this manipulation to the luminance channel, the image may be sharpened by applying a single convolution to the luminance channel rather than convolving each of the red, green, and blue image signals by separate sharpening kernels. Using this approach, the efficiency of the image processing system is improved. While this process sharpens the luminance channel within the image, it does not necessarily improve the reconstruction of edge information and like the previous patent application, it does not anticipate that such a method might be significantly more beneficial when provided in a display having more high luminance subpixels than pixels or when applied in a display system having not only red, green, and blue light-emitting elements, but also additional light-emitting elements such that the number of convolutions might be reduced to one fourth or even more.
There is a need, therefore, for an improved image processing method and associated arrangements of light-emitting elements for improving the apparent resolution of displays wherein the arrangement of light-emitting elements contain more high luminance light-emitting elements than pixels. Particularly, such a method should provide a means of providing a higher image quality when rendering an image to an arrangement of red, green, blue, and at least one additional light-emitting element.